ABSTRACT
Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm, and the details of model training. We aim at establishing a firm link between communication research tasks and the corresponding state-of-the-art in natural language processing research by systematically comparing the performance of different automatic text analysis approaches. We do this for a challenging task - stance detection of opinions on policy measures to tackle the COVID-19 pandemic in Germany voiced on Twitter. Our results add evidence that pre-trained language models such as BERT outperform feature-based and other neural network approaches. Yet, the gains one can achieve differ greatly depending on the specific merits of pre-training (i.e., use of different language models). Adding to the robustness of our conclusions, we run a generalizability check with a different use case in terms of language and topic. Additionally, we illustrate how the amount and quality of training data affect model performance pointing to potential compensation effects. Based on our results, we derive important practical recommendations for setting up such SML tools to study communication texts.
ABSTRACT
Many studies show a positive relationship between media use and protest par-ticipation. Yet, the picture becomes less clear-cut for different types of media (traditional vs. social media). Thus, the mechanisms underlying these mobilizing media effects remain vague. This paper attempts to address this research gap by looking more closely at media -related factors (evaluation of media coverage about one's protest group) and relating them to participatory predictors (political efficacy). Based on a survey of activists (N = 132) from randomly selected protest groups in Germany, we analyze both media perceptions and political efficacy as mediators between the use of different information sources and protest behavior. Path analyses showed that using traditional news media and social media was differently related to collective action: In contrast to social media, using traditional news media were related to a positive impression how the media covered one's own group. This perception of the media coverage was linked to greater political efficacy and, ulti-mately more protest behavior offline. Yet, the model paths differed for activists from dif-ferent groups, e.g. environmental groups vs. so-called "concerned citizens" who protested against the governmental measures to contain the coronavirus pandemic.
ABSTRACT
This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement - in terms of inter-annotator agreement (+.14 Fleiss' kappa) and annotation quality - compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.(1)